Bayesian Ying Yang System and Theory as A Uni ed Statistical Learning Approach II From Unsupervised Learning to Supervised Learning and Temporal Modeling

A uni ed statistical learning approach called Bayesian Ying Yang BYY system and theory has been developed by the present author in recent years In a sister paper this BYY system and theory has been shown to function as a general theory for unsupervised learning and its extension called semi unsupervised learning such that not only sev eral existing popular unsupervised learning approaches are included as special cases but also a number of new theories and models are provided for unsupervised pattern recognition and cluster analysis factorial en coding data dimension reduction and independent component analysis In this paper the basic system and theory in is further theoretically justi ed and extended into a general system and theory with a gen eral implementation technique such that not only those results are kept as special cases still but also it works for supervised learning and temporal modeling on parameter learning regularization structural scale or complexity selection and architecture design Particularly temporal modeling and regression based on Hidden Markov Model HMM and the linear and nonlinear state space model are discussed in detail with an adaptive algorithm proposed for various speci c variants of HMM model and state space models Moreover the criteria for deciding the number of hidden states in HMM and the order of state space are also proposed In another sister paper of this proceeding several speci c models and algorithms as well as model selection criteria will be given for dependence reduction data dimension reduction independent com ponent analysis supervised classi cation and regression In addition the Supported by the HK RGC Earmarked Grants CUHK E and CUHK E and by Ho Sin Hang Education Endowment Fund for Project HSH The basic ideas of the BYY learning in my previous papers started the rst year of my returning to HK As HK in transition to China this work was in transition to its current shape The preliminary version of this paper and its sister papers are all completed in the rst month that HK returned to China and thus I formally returned to my motherland as well I would like to use this work as a memory of this historic event relation of the BYY learning system and theory to a number of existing learning models and theories has been discussed in Basic Bayesian Ying Yang System and Theory As shown in Fig the BYY system consists of seven components The rst four components form the core The other three surrounding components are added for the purposes of supervised learning The core itself functions as a general framework for unsupervised learning as shown in In this section we understand the basic idea of the core As shown in unsupervised perception tasks can be summarized into the problem of estimating the joint distribution p x y of the observable pattern x in the observable space X and its representation pattern y in an invisible space Y In the Bayesian framework we have two complementary representations p x y p yjx p x and p x y p xjy p y We use two sets of models M fMyjx Mxg and M fMxjy Myg to implement each of the two representations pM pM x y pMyjx yjx pMx x pM pM x y pMxjy xjy pMy y We call Mx a Yang visible model which describes p x in the visible domain X andMy a Ying invisible model which describes p y in the invisible domain Y Also we call the passage Myjx for the ow x y a Yang male passage since it performs the task of transferring a pattern a real body into a code a seed We call a passage Mxjy for the ow y x a Ying female passage since it performs the task of generating a pattern a real body from a code a seed Together we have a YANG machine M to implement pM x y and a YING machine M to implement pM x y A pair of YING YANG machines is called a YING YANG pair or a Bayesian YING YANG system Such a formalization compliments to a famous Chinese ancient philosophy that every entity in the universe involves the interaction between YING and YANG The task of specifying a Ying Yang system is called learning in a broad sense which consists of the following four levels of speci cations Item According to the nature of the perception task the Representation Domain Y and Its Complexity k are designed For example we have either y R or a binary vector y y y k T y j f g Item Based on the given set of training samples some previous knowl edge assumption and heuristics Architecture Design is made by specifying the architectures of four components pMx x pMyjx yjx pMxjy xjy and pMy y First with a given set Dx fxig N i from an original density p x pMx x is xed at some parametric or nonparametric empirical density estimation of p x It should be Yin in the Mainland Chinese spelling system However I prefer to use Ying for the beauty of symmetry Furthermore strictly speaking we should use P u to replace p u when the corresponding random variable u is discrete However we simply use p for both the cases Readers may identify the di erence according to whether the involved variable is real or discrete Representation Space Y Symbols, Integers, Binary Codes Reals Input pattern space X Output Action Space Z 1